Single-chip sensor multi-function imaging

ABSTRACT

Mixed mode imaging is implemented using a single-chip image capture sensor with a color filter array. The single-chip image capture sensor captures a frame including a first set of pixel data and a second set of pixel data. The first set of pixel data includes a first combined scene, and the second set of pixel data includes a second combined scene. The first combined scene is a first weighted combination of a fluorescence scene component and a visible scene component due to the leakage of a color filter array. The second combined scene includes a second weighted combination of the fluorescence scene component and the visible scene component. Two display scene components are extracted from the captured pixel data in the frame and presented on a display unit.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to and the benefit of:

-   -   U.S. Provisional Application No. 61/646,710 filed May 14, 2012        entitled “Single-chip Sensor Multi-Function Imaging,” naming as        inventors, Wenyi Zhao, Jeffrey DiCarlo, and Ian McDowall; and    -   U.S. Provisional Application No. 61/646,727 filed May 14, 2012        entitled “Single-chip Sensor Multi-Function Imaging,” naming as        inventors, Jeffrey DiCarlo, Ian McDowall, and Wenyi Zhao, both        of which are incorporated herein by reference in their entirety.

This application is related to the following concurrently filed U.S.patent application Ser. No. 13/893,536, (disclosing “SINGLE-CHIP SENSORMULTI-FUNCTION IMAGING”) which is incorporated by reference:

BACKGROUND

1. Field of Invention

Aspects of this invention are related to endoscopic imaging, and aremore particularly related to blending visible and alternate images so asto provide an enhanced real-time video display for a surgeon.

2. Related Art

The da Vinci® Surgical Systems, commercialized by Intuitive Surgical,Inc., Sunnyvale, Calif., are minimally invasive teleoperated surgicalsystems that offer patients many benefits, such as reduced trauma to thebody, faster recovery and shorter hospital stay. One key component of ada Vinci® Surgical System (e.g., the model IS3000, da Vinci® Si HD) is acapability to provide two-channel (i.e., left and right) video captureand display of visible images to provide stereoscopic viewing for thesurgeon. Such electronic stereoscopic imaging systems may output highdefinition video images to the surgeon, and may allow features such aszoom to provide a “magnified” view that allows the surgeon to identifyspecific tissue types and characteristics, as well as to work withincreased precision.

In a typical surgical field, however, certain tissue types are difficultto identify, or tissue of interest may be at least partially obscured byother tissue. Thus, different imaging modes have proven useful forsurgeons during surgery. There are systems with single imaging mode andother systems with multiple imaging modes that provide for examplevisible scenes in one imaging mode and fluorescence scenes in anotherimaging mode.

SUMMARY

In mixed mode imaging, a unique problem was discovered when using asingle-chip image capture sensor with a color filter array. The lightfrom the surgical site is passed through color filters in a color filterarray and then captured as a frame of pixel data by the single-chipimage capture sensor. Unfortunately, the color filters in the colorfilter array allow leakage between adjacent color components in thevisible spectrum and do not block wavelengths outside the visiblespectrum. Thus, previous techniques used to implement mixed mode imagingdo not work when a single-chip image capture sensor with a color filterarray is used because imaging modes are not restricted to single colorchannels; all channels see a combination of the multiple image modes.

However, in one aspect, mixed mode imaging is implemented using asingle-chip image capture sensor with a color filter array. Thesingle-chip image capture sensor captures a scene of a surgical sitethat includes a fluorescence scene component and a visible scenecomponent in a frame of pixel data.

In one aspect, the frame includes a first plurality of pixel data and asecond plurality of pixel data, sometimes referred to as a first set ofpixel data and a second set of pixel data. The first plurality of pixeldata includes a first combined scene, and the second plurality of pixeldata includes a second combined scene. The first combined scene, in oneaspect, is a first weighted combination of a fluorescence scenecomponent and a visible scene component due to the leakage of the colorfilter array. The second combined scene includes a second weightedcombination of the fluorescence scene component and the visible scenecomponent.

The surgical system also includes a scene processing module. The sceneprocessing module extracts a plurality of display scene components fromthe frame of pixel data. The plurality of display scene components ispresented on a display unit.

More specifically, in one aspect, the scene processing module receivesthe first plurality of pixel data including the first combined scene andthe second plurality of pixel data including the second combined scene.A display fluorescence scene component is extracted from the first andsecond combined scenes, and a display visible scene component isextracted from the first and second combined scenes. The displayfluorescence scene component corresponds to the fluorescence scenecomponent, while the display visible scene component corresponds to thevisible scene component. The scene processing module generates aplurality of weighted combinations of the display fluorescence scenecomponent and the display visible scene component.

A display unit, in the surgical system, is connected to the sceneprocessing module. The display unit receives the plurality of weightedcombinations, and generates from the plurality of weighted combinationsa displayed scene. The displayed scene includes a highlighted scenecomponent corresponding to the fluorescence scene component and areduced color scene component corresponding to the visible scenecomponent.

The reduced color scene component combined with the fluorescence scenecomponent provides a scene of a surgical site with pathology informationand/or anatomic information highlighted for the surgeon. The highlightedfluorescence scene component identifies tissue of clinical interest. Thecombination of a reduced color scene component with a fluorescence scenecomponent is one example of mixed-mode imaging.

The surgical system also includes an illuminator providing at least twoillumination components. When one of the illumination components is afluorescence excitation illumination component, other illuminationcomponents include less than all visible color components of whitelight. The at least two illumination components are provided at the sametime. In one aspect, the illuminator includes a fluorescence excitationillumination source and a visible color component illumination source.

In one aspect, the scene processing module includes a demosaic modulecoupled to the image capture unit to receive, in a first color channel,the first plurality of pixel data including the first combined scene.The demosaic module demosaics the first plurality of pixel data toobtain a first plurality of image pixel data comprising the firstcombined scene. The demosaic module also is coupled to the image captureunit to receive, in a second color channel, the second plurality ofpixel data including the second combined scene. The demosaic moduledemosaics the second plurality of pixel data to obtain a secondplurality of image pixel data including the second combined scene.

In another aspect, the scene processing module includes a demosaicmodule coupled to the image capture unit to receive, in a first colorchannel, the first plurality of pixel data comprising the first combinedscene, and to receive, in a third color channel, a third plurality ofpixel data comprising a third combined scene. The demosaic moduledemosaics the first and third pluralities of pixel data as a singlecolor channel to obtain a first plurality of image pixel data comprisinga fourth combined scene. The demosaic module also is coupled to theimage capture unit to receive, in a second color channel, the secondplurality of pixel data including the second combined scene. Thedemosaic module demosaics the second plurality of pixel data to obtain asecond plurality of image pixel data including the second combinedscene.

The scene processing module also includes a scene component generator.The scene component generator receives a first plurality of image pixeldata of a first color component and a second plurality of image pixeldata of a second color component. The first plurality of image pixeldata includes one of the first combined scene and the fourth combinedscene. The second plurality of image pixel data includes the secondcombined scene. The scene component generator performs the extraction ofthe display fluorescence scene component. The display fluorescence scenecomponent comprises a first linear weighted combination of the firstplurality of image pixel data and the second plurality of image pixeldata. The scene component generator also performs the extraction of thedisplay visible scene component. The display visible scene componentcomprises a second linear weighted combination of the first plurality ofimage pixel data and the second plurality of image pixel data. Thedisplay fluorescence scene component is different from the displayvisible scene component. The display fluorescence scene componentcorresponds to the fluorescence scene component, and the display visiblescene component corresponds to the visible scene component.

The surgical system performs a method including receiving a frame ofpixel data captured by a single-chip image capture sensor. Thesingle-chip image capture sensor includes a color filter array. Thepixel data comprises a scene of a surgical site that includes afluorescence scene component and a visible scene component.

In one aspect, the frame includes a first set of pixel data including afirst combined scene, and a second set of pixel data including a secondcombined scene. The first combined scene, in one aspect, is a firstweighted combination of the fluorescence scene component and the visiblescene component. The second combined scene includes a second weightedcombination of the fluorescence scene component and the visible scenecomponent. The method extracts a display fluorescence scene component, adisplay fluorescence scene component, from the first combined scene andextracts a display visible scene component, a display visible scenecomponent, from the second combined scene. Then, the method generates aplurality of weighted combinations of the display fluorescence scenecomponent and the display visible scene component. Finally, the methodgenerates from the plurality of weighted combinations a displayed sceneincluding a highlighted fluorescence component corresponding to thefluorescence scene component and a reduced color scene componentcorresponding to the visible scene component.

In one aspect, the method includes illuminating a surgical site with atleast two illumination components. When one of the illuminationcomponents is a fluorescence excitation illumination component, otherillumination components include less than all visible color componentsof white light. The at least two illumination components are provided atthe same time.

The method further includes demosaicing the first plurality of pixeldata comprising the first combined scene to obtain a first plurality ofimage pixel data comprising the first combined scene. The secondplurality of pixel data comprising the second combined scene is alsodemosaiced to obtain a second plurality of image pixel data comprisingthe second combined scene.

In another aspect, the method includes demosaicing the first pluralityof pixel data comprising the first combined scene and a third pluralityof pixel data comprising a third combined scene as a single colorchannel to obtain a first plurality of image pixel data comprising afourth combined scene. In this aspect, the second plurality of pixeldata comprising the second combined scene also is demosaiced to obtain asecond plurality of image pixel data comprising the second combinedscene.

The method still further includes receiving a first plurality of imagepixel data of a first color component and a second plurality of imagepixel data of a second color component. The first plurality of imagepixel data comprises the first combined scene, and the second pluralityof image pixel data comprises the second combined scene. The methodperforms the extraction of the display fluorescence scene component. Thedisplay fluorescence scene component comprises a first linear weightedcombination of the first plurality of image pixel data and the secondplurality of image pixel data. The method also performs the extractionof the display visible scene component. The display visible scenecomprises a second linear weighted combination of the first plurality ofimage pixel data and the second plurality of image pixel data. Thedisplay fluorescence scene component is different from the displayvisible scene component. The display fluorescence scene componentcorresponds to the fluorescence scene component, and the display visiblescene component corresponds to the visible scene component.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high level diagrammatic view of a minimally invasiveteleoperated surgical system including an augmented stereoscopicvisualization system using a single-chip image capture sensor with acolor filter array.

FIG. 2 is a schematic view that illustrates hardware and software (imageprocessing and user interface) aspects of augmented stereoscopicvisualization system.

FIG. 3 is process flow diagram of a process performed using, forexample, the augmented stereoscopic visualization system of theminimally invasive teleoperated surgical system of FIG. 1.

FIG. 4 illustrates one aspect of the timing, synchronization, andcapture of frames in the system in FIGS. 2 and 3.

FIG. 5 is a more detailed block diagram of the augmented colorcorrection module of FIG. 2.

FIG. 6A is a representation of a normal color stereoscopic imageobtained using the system of FIG. 2.

FIG. 6B is a representation of a reduced color stereoscopic imageobtained using the system of FIG. 2 with a highlighted fluorescenceimage.

FIG. 7 is a more detailed block diagram of the scene processing moduleof FIG. 5.

FIG. 8A illustrates input data to and the output data from the demosaicmodule of FIG. 7 in a first aspect.

FIG. 8B illustrates input data to and the output data from the demosaicmodule of FIG. 7 in a second aspect.

FIG. 9 is a more detailed block diagram of the display color componentgenerator of FIG. 5.

In the drawings, the first digit of a reference number indicates thefigure in which the element with that reference number first appearedfor single digit figure numbers.

DETAILED DESCRIPTION

As used herein, electronic stereoscopic imaging includes the use of twoimaging channels (i.e., one channel for left side scenes and anotherchannel for right side scenes).

As used herein, a stereoscopic endoscope includes two channels (e.g.,channels for left and right images) for transporting light from anobject such as tissue to be imaged. The light transported in eachchannel represents a different view (stereoscopic left or right) of ascene in the surgical field. Without loss of generality orapplicability, the aspects described more completely below also could beused in the context of a field sequential stereo acquisition systemand/or a field sequential display system.

As used herein, an illumination channel provides illumination to tissuefrom an illumination source located away from an image capture unit(e.g., away from the distal end of an endoscope), or an illuminationsource located at or near the image capture unit (e.g., one or morelight emitting diodes (LEDs) at or near the distal end of an endoscope).

As used herein, scenes captured in the visible electromagnetic radiationspectrum are referred to as visible scenes.

As used herein, white light is visible white light that is made up ofthree (or more) visible color components, e.g., a red visible colorcomponent, a green visible color component, and a blue visible colorcomponent. If the visible color components are provided by anilluminator, the visible color components are referred to as visiblecolor illumination components. White light may also refer to a morecontinuous spectrum in the visible spectrum as one might see from aheated tungsten filament or xenon lamp, for example.

As used herein, a monochromatic scene that is generated using anilluminator that provides less than all of a plurality of visible colorilluminations components of white light is referred to as a reducedcolor scene. If the reduced color scene is a part of a scene thatincludes another scene component, e.g., a fluorescence scene component,the reduced color scene is referred to as a reduced color scenecomponent.

As used herein, a visible scene component includes a visible colorcomponent.

As used herein, “first,” “second,” and “third” are adjectives used todistinguish between color components. Thus, “first,” “second,” and“third” are not intended to imply any ordering of the color componentswithin the visible wavelength spectrum.

As used herein, a single-chip image capture sensor includes a singleintegral semiconductor device on which all the color components in whitelight are captured. The photosensors on the chip detect light intensitywith little or no wavelength specificity. Thus, the single-chip imagecapture sensor also includes a color filter array. The color filterarray filters the incoming light by wavelength range so that differentsets of photosensors capture different color components.

As used herein, pixels refer to photosensors on the single-chip imagecapture sensor. The pixels capture pixel data. A set of pixel datacaptured for a color component is interpolated, referred to herein asdemosaiced, to generate a plurality of image pixel data for that colorcomponent. Thus, a pixel and an image pixel, as used herein, aredifferent entities.

As used herein, scene components captured by an image capture sensor asthe result of fluorescence are referred to as acquired fluorescencescene components, and sometimes simply as fluorescence scene components.There are various fluorescence imaging modalities. Fluorescence mayresult from natural tissue fluorescence, or the use of, for example,injectable dyes, fluorescent proteins, or fluorescent tagged antibodies.Fluorescence may result from, for example, excitation by laser or otherenergy source. In such configurations, it is understood that a notchfilter is used to block the excitation wavelength that enters theendoscope. Fluorescence scene components can provide vital in vivopatient information that is critical for surgery, such as pathologyinformation (e.g., fluorescing tumors) or anatomic information (e.g.,fluorescing tagged tendons).

Aspects of this invention augment the stereoscopic video capturing andviewing capability of a minimally invasive surgical system, e.g., the daVinci® minimally invasive teleoperated surgical system commercialized byIntuitive Surgical, Inc. of Sunnyvale, Calif., by incorporating bothstereoscopic normal visible scenes, and alternatively stereoscopicreduced color scene components combined with fluorescence scenecomponents. (da Vinci® is a registered trademark of Intuitive SurgicalOperations, Inc. of Sunnyvale, Calif.) A stereoscopic reduced colorscene component combined with a fluorescence scene component provides astereoscopic image of a surgical site with pathology information and/oranatomic information highlighted for the surgeon. The highlightedfluorescence scene component identifies tissue of clinical interest. Thecombination of a reduced color scene component with a fluorescence scenecomponent is one example of mixed-mode imaging.

The stereoscopic reduced color scene component combined with thefluorescence scene component is provided in real time to a surgeonperforming a surgical operation using a minimally invasive teleoperatedsurgical system. A minimally invasive teleoperated surgical system is anexample of a robotic surgical system. Sequential acquisition approaches(also known as time slicing) incur a delay associated with capturing astereoscopic image in one frame and the fluorescence image in anotherframe and then using the two frames taken at different points in time togenerate a single frame that is displayed for the surgeon. Hence, thememory and processing requirements of the system described herein arereduced with respect to systems that utilize time slicing to superimposea fluorescence scene component on a stereoscopic color visible scenecomponent.

The stereoscopic reduced color scene component is formed using anilluminator that provides less than all the plurality of visible colorillumination components that make white light and so color informationin the reduced color scene component is lost but there is little or noloss in detail. The stereoscopic reduced color scene component issufficient to identify anatomy, tissue landmarks, and surgicalinstruments so that this image allows safe manipulation of the surgicalinstruments. With the reduced color scene component, there is no loss incontrast of the fluorescence scene component due to interference by avisible color illumination component.

In one aspect, the fluorescence scene component is overlaid onto thereduced color scene component and color enhanced to provide improvedinformation content regarding the surgical site that reduces the risk ofinjury to the patient and that improves surgical efficiency. Thiscombination of a stereoscopic reduced color scene component and ahighlighted fluorescence scene component provides benefits including,but not limited to, allowing a surgeon in real-time to identify positivetumor margins for diseased tissue excision and to identify other tissue,e.g., tendons, so as to avoid unnecessarily cutting that tissue.

The combination of the reduced color scene components and thefluorescence scene components may be continuously displayed to thesurgeon. Alternatively, the overlay of the two scene components may betoggled on and off (e.g., by using a foot pedal or by double-clickingmaster finger grips on the da Vinci® Surgical System surgeon's console).

Herein, an example of mixed mode imaging is simultaneously displayingboth a reduced color scene of a white-light scene and a fluorescencescene that are optically aligned. Typically, the reduced color scene isrequired to carry out surgery while the fluorescence scene helps asurgeon to locate blood vessels and cancer tissues and margins, asindicated above.

Mixed imaging modes, sometimes referred to as augmented modes, are ofgreat interest to a surgeon since the surgeon can carry out the completecycle of diagnostics, surgery, and monitoring on the fly. It is quitecommon that one color camera and one near infrared camera are integratedto operate under such modes.

To reduce the integration complexity associated with the integration oftwo cameras, an image capture system based on a three-chip sensor hasbeen used for mixed mode imaging. With the three-chip sensor, forexample, green illumination, blue illumination, and near infrared laserillumination are simultaneously used to obtain a black and white versionof a white image from the green and blue color channels and afluorescence scene from the red color channel. Unfortunately, this isnot a solution for an inexpensive single-chip image capture sensor, asdescribed more completely below.

There are other possible solutions to mixed mode imaging such as thesequential (color or wavelength) imaging systems with a mono-chromesensor and tunable illuminations. The main issue of these sequentialimaging systems is the motion-induced color fringe between images takenat different times under different illuminations.

In mixed mode imaging using a single-chip image capture sensor with acolor filter array to capture information, the light from the surgicalsite is passed through filters in the color filter array and thencaptured as pixel data by the single-chip image capture sensor. Eachfilter in the color filter array is a band pass filter. A band passfilter passes light having wavelengths within a certain range and blockspassage of wavelengths outside that range. Thus, the component of thelight, i.e., the wavelengths of light, from surgical site captured by apixel is determined by the color of the filter for that pixel in thecolor filter array.

Many different types of color filter arrays are known. A commonly usedcolor filter array is a Bayer color filter array. The Bayer color filterarray has three different color filters. For example, with a color modelthat has red, green, and blue color components, the Bayer color filterarray has red filters, blue filters, and green filters. As noted above,each filter is a bandpass filter. A green filter passes green light andblocks red and blue light. Thus, a pixel with a green filter capturesgreen light. A pixel with a red filter captures red light, and a pixelwith a blue filter captures blue light.

Typically, the bandpass filters are made using dyes. The dyes workreasonably well in the visible spectrum, but do not block light that isoutside the visible spectrum. For example, if fluorescence is in theinfrared spectrum, the fluorescence passes through each of the red,green, and blue filters and is captured by the corresponding pixels.Thus, if green color component illumination in conjunction withfluorescence excitation illumination that excites fluorescence in theinfrared spectrum is used, the red and blue pixels capture thefluorescence, while the green pixel captures a combination of thereflected green light and the fluorescence.

Also, typically there is some leakage of adjacent visible colorcomponents though a filter in the Bayer filter array. Thus, a greenpixel may capture some red light and some blue light in addition to thegreen light. A red pixel may capture some green light in addition to thered light, and a blue pixel may capture some green light in addition tothe blue light.

Thus, in the augmented mode, when green light illumination is used inconjunction with florescence excitation illumination that excitesinfrared fluorescence, the surgical site scene includes reflected greenlight, which is an example of a visible scene component, andfluorescence, which is a fluorescence scene component. Each of the red,green and blue color filters in the color filter array passes thefluorescence so that all pixels capture the fluorescence scenecomponent. The red color filter also passes a portion of the reflectedgreen light, sometimes referred to as leakage, and so the pixelsassociated with the red color filter capture a first combination of aportion of the reflected green light and the fluorescence. The greencolor filter passes the reflected green light and so the pixelsassociated with the green color filter capture a second combination ofthe reflected green light and the fluorescence. The blue color filteralso passes a portion of the reflected green light and so the pixelsassociated with the blue color filter capture a third combination of aportion of the reflected green light and the fluorescence.

In more general terms, the pixels for a first color component capture acombined scene that is a first weighted combination of the fluorescencescene component and a visible color component scene component. Thepixels for a second color component capture a combined scene that is asecond weighted combination of the fluorescence scene component and avisible color component scene. The pixels for a third color componentcapture a combined scene that is a third weighted combination of thefluorescence scene component and a visible color component scene. Here,weighted combination is used to convey the concept that the lightcaptured by a set of pixels is a combination of the visible colorcomponent scene and the fluorescence scene component and the particularweighted combination is determined by the characteristics of the colorfilter for a color component.

As described more completely below, mixed mode imaging is implementedusing a single-chip image capture sensor that includes a color filterarray. The characteristics of the color filter array, as describedabove, and the resulting light that is captured are taken into account,and a plurality of display scene components are extracted from thecaptured pixel data and presented on a display unit. In the followingdescription, a fluorescence scene component is used as an example of afirst scene component, and a visible color component scene is used as anexample of a second scene component. In this example, a combined sceneis a combination of the first scene component and the second scenecomponent. As described above, a first set of pixels in the single-chipimage capture sensor captures the first combined scene, and a second setof pixels in the single-chip image capture sensor captures the secondcombined scene. The pixel data is captured in the same frame as opposedto in two different frames in the previous sequential processingtechniques.

Also, in the following examples, fluorescence excitation illumination isused simultaneously with a single visible color illumination component.The fluorescence excitation illumination excites fluorescence havingwavelengths outside the visible spectrum.

The use of a single visible color illumination component also isillustrative and is not intended to be limiting. The aspects describedmore completely below work with illumination that includes less than allthe color components of white light. Also, the fluorescence does nothave to be outside the visible spectrum. For example, if greenillumination is used, and the fluorescence is in the blue spectrum suchthat some of the fluorescence leaks through the green color filter, thefollowing aspects are applicable.

FIG. 1 is a high level diagrammatic view of a minimally-invasiveteleoperated surgical system 100, for example, the da Vinci® SurgicalSystem, including a stereoscopic visualization system. In this example,a surgeon, using a surgeon's console 114, remotely manipulates anendoscope 112 mounted on a robotic manipulator arm 113 that in turn ismounted on cart 110. There are other parts, cables etc. associated withthe da Vinci® Surgical System, but these are not illustrated in FIG. 1to avoid detracting from the disclosure. Further information regardingminimally invasive surgical systems may be found for example in U.S.patent application Ser. No. 11/762,165 (filed Jun. 23, 2007; disclosingMinimally Invasive Surgical System), U.S. Pat. No. 6,837,883 B2 (filedOct. 5, 2001; disclosing Arm Cart for Telerobotic Surgical System), andU.S. Pat. No. 6,331,181 (filed Dec. 28, 2001; disclosing SurgicalRobotic Tools, Data Architecture, and Use), all of which areincorporated herein by reference.

As explained more completely below, an illumination system (not shown),sometimes referred to as an illuminator, is coupled to endoscope 112. Inone aspect, the illumination system selectively provides one of (a)white light illumination and (b) less than all the visible colorillumination components of white light and at least one fluorescenceexcitation illumination component. The light from the illuminationsystem is coupled to at least one illumination channel in endoscope 112,in one aspect. The light passes through at least one illuminationchannel in endoscope 112 and illuminates tissue 103 of a patient 111.

Endoscope 112 also includes, in one aspect, two stereoscopic opticalchannels, e.g., a left optical channel and a right optical channel, forpassing light from the tissue, e.g., reflected white light or thereflected light from the visible color illumination component(s) andfluorescence. The white light reflected from tissue 103 is captured as anormal acquired visible color stereoscopic scene by image capture system120. However, when the surgeon wants to see an augmented scene, thesurgeon changes the viewing mode to an augmented viewing mode.

In the augmented viewing mode, the illumination source provides lessthan all of the visible color illumination components of white light.For example, if three visible color illumination components of whitelight are used, at most two visible color illumination components areprovided by the illumination source. Thus, in this aspect of theaugmented viewing mode, tissue 103 is illuminated with one or twovisible color illumination components, e.g., with less than all theplurality of visible color illumination components of white light, and afluorescence excitation illumination component.

Each of the left and right channels in image capture system 120 includesa single-chip image capture sensor with a color filter array. Thus, inthe augmented viewing mode when tissue 103 is illuminated with one ortwo visible color illumination components and a fluorescence excitationillumination component, each image capture unit captures a scene of asurgical site that includes a fluorescence scene component and a visiblescene component in a frame of pixel data. A first plurality of pixelscaptures a first combined scene 125 in a frame 121, e.g., captures afirst combined scene including a first weighted combination of thefluorescence scene component and the visible color component scene. Asecond plurality of pixels captures a second combined scene 126 in frame121, e.g., captures a second combined scene including a second weightedcombination of the fluorescence scene component and the visible colorcomponent scene.

In more general terms, the single-chip image capture sensor with a colorfilter array includes a first plurality of pixels that captures a firstcombined scene 125 in a frame of pixel data 121. First combined scene125 is a first weighted combination of a first scene component of thesurgical site and a second scene component of the surgical site. Thesingle-chip image capture sensor also includes a second plurality ofpixels that capture a second combined scene 126 in frame 121. Secondcombined scene 126 is a second weighted combination of the first scenecomponent of the surgical site and the second scene component of thesurgical site. The second scene component is different from the firstscene component. Also, the first weighted combination is different fromthe second weighted combination.

In this aspect, image capture system 120 is a conventional single-chipimage capture system with a color filter array except any filter orfilters that would block the fluorescence are removed. A filter orfilters may be used to block capture of any direct light or reflectedlight from the fluorescence excitation source or sources.

Display image controller 130 receives the acquired pixel data from imagecapture system 120. When the acquired pixel data is normal visible colorstereoscopic scenes, a color correction module 136 in an imagecorrection module 135 of display image controller 130 processes theacquired pixel data including color correction to generate colorcorrected visible full color stereoscopic scenes. The color correctedvisible full color stereoscopic scenes are sent to the stereoscopicviewer on surgeon's console 114, which displays the stereoscopic scenes.

Similarly, in the augmented viewing mode, display image controller 130receives the acquired pixel data from image capture system 120. Forexample, for a three-component color space, if the illumination sourceprovides one visible color illumination component i.e., a first visiblecolor illumination component, and a fluorescence excitation illuminationcomponent, three sets of pixel data are acquired. A first set of theacquired pixel data is a first combined scene that includes a firstweighted combination of the acquired visible color component scene andthe acquired fluorescence scene component. A second set of the acquiredpixel data is a second combined scene that includes a second weightedcombination of the acquired visible color component scene and theacquired fluorescence scene component. A third set of acquired pixeldata is a third combined scene that includes a third weightedcombination of the acquired visible color component scene and theacquired fluorescence scene component. Display image controller 130receives the three sets of acquired pixel data that are in a singleframe.

In the augmented viewing mode, an augmented color correction module 137in image correction module 135 of display image controller 130 uses anaugmented color correction process to process the acquired pixel data ineach of the left and right channels of image capture system 120. Theaugmented image correction process generates a display visible colorscene component and a display fluorescence scene component in each ofthe left and right channels.

In one aspect, the augmented color correction module in display imagecontroller 130, as explained more completely below, provides a linearweighted combination of the display visible component scene and thedisplay fluorescence scene component to each color channel of thedisplay unit in surgeon's console 114. The display unit displays adisplay scene that includes a reduced color scene component and afluorescence scene component. The fluorescence scene component ishighlighted on the display unit because pixels for a color component inthe display that receive a part of the fluorescence scene component alsoreceive a part of the reduced color scene component are so are brighterrelative to neighboring pixels of the reduced color scene component thatreceive only a part of the reduced color scene component.

FIG. 2 is a more detailed illustration of the aspects of one example ofminimally invasive surgical system 100 of FIG. 1. FIG. 3 is a high-levelprocess flow diagram for the operation of the system in FIG. 2, whileFIG. 4 is a timing diagram for the illumination of a surgical site andthe capture of frames in the system of FIG. 2.

In the embodiment of FIG. 2, minimally invasive surgical system 200includes an illuminator that is combination light source 210.Combination light source 210 includes a visible color component source211 and a fluorescence excitation source 212, in this aspect. Theparticular implementation of sources 211 and 212 is not critical so longas combination light source 210 has the capabilities described morecompletely below.

Combination light source 210 is used in conjunction with at least oneillumination channel in a stereoscopic endoscope 201 to illuminatesurgical site 203 in an ILLUMINATE TISSUE process 302 (FIG. 3). In thisexample, combination light source 210 has two modes of operation: anormal viewing mode and an augmented viewing mode.

In the normal viewing mode, visible color component source 211 providesillumination that illuminates surgical site 203 in white light, i.e.,all the visible color component illumination sources in source 211 areused. Fluorescence excitation source 212 is not used in the normalviewing mode.

In the augmented viewing mode, visible color component source 211provides less than all the visible color components needed to illuminatesurgical site 203 in white light, e.g., one or more of the visible colorcomponents of white light are not included in the illumination. In someaspects, for example, it may be possible to use all the visible colorillumination components of white light in the augmented viewing mode,but include trivial illumination from one or more of the visible colorillumination components and augmented viewing mode illumination from theremaining visible color illumination component or components. Trivialillumination means that the illumination provided by the visible colorcomponent illumination source is so low that when reflected light fromthe trivial illumination and florescence are acquired together, theacquired trivial illumination data does not degrade the acquiredfluorescence data. Thus, providing trivial illumination for the one ormore visible color illumination components is effectively the same asilluminating the tissue with less than all the visible color componentsof white light.

In one aspect, three visible color components make up white lightillumination, i.e., white light includes a first visible color componentC1, a second visible color component C2, and a third visible colorcomponent C3. Each of the three visible color components C1, C2, C3 is adifferent visible color component, e.g., a red component, a greencomponent, and a blue component. The use of three visible colorcomponents C1, C2, C3 to make up white light illumination isillustrative of a plurality of such components and is not intended to belimiting.

In the augmented viewing mode, fluorescence excitation source 212provides a fluorescence excitation illumination component that excitesfluorescence from surgical site 203. For example, narrow band light fromfluorescence excitation source 212 is used to excite tissue-specificfluorophores so that fluorescence scenes of specific tissue withinsurgical site 203 are captured.

In the augmented viewing mode, the number of visible color illuminationcomponents provided by visible color component source 211 depends on:the number of different fluorescence scene components captured; and thenumber of color components used to make white light. In one aspect,where three visible color illumination components C1, C2, C3 are used tomake white light and a single-chip image capture sensor with a colorfilter array is used in each of the left and right channels of imagecapture system 220, one visible color illumination component is providedby visible color component source 211 and fluorescence excitation source212 provides fluorescence excitation illumination.

In one aspect, visible color component source 211 includes a source foreach of the different visible color illumination components in theplurality of visible color illumination components of white light. For ared-green-blue implementation, in one example, the sources are lightemitting diodes (LEDs), a red LED, two green LEDs and a blue LED. Table1 gives the range of output wavelengths for each of the LEDs used inthis example.

TABLE 1 Visible Color Illumination Component Wavelength Red 670nanometers (nm) Green 1 555 nm Green 2 532 nm Blue 450 nm

The use of LEDs in visible color component source 211 is illustrativeonly and is not intended to be limiting. Visible color component source211 could also be implemented with multiple laser sources instead ofLEDs for example. Alternatively, visible color component source 211could use a Xenon lamp with an elliptic back reflector and a band passfilter coating to create broadband white illumination light for visibleimages. The use of a Xenon lamp also is illustrative only and is notintended to be limiting. For example, a high pressure mercury arc lamp,other arc lamps, or other broadband light sources may be used. Toeliminate one or more visible color illumination components from such asource in the augmented viewing mode, bandpass filters, prisms etc.could be incorporated in combination light source 210.

The use of a light source that is removed from endoscope 201 is alsoillustrative and not intended to be limiting. In some aspects, the lightsource could be mounted on the distal end of endoscope 201, for example.

Also, in the augmented viewing mode for a fluorescence excitationwavelength occurs outside the visible spectrum (e.g., in the nearinfrared (NIR)), a laser module (or other energy source, such as alight-emitting diode or filtered white light) is used as fluorescenceexcitation source 212. Thus, in one aspect, fluorescence is triggered bylight from a laser module in fluorescence excitation source 212. Forexample, the FDA approved fluorescent dye Indocyanine Green has anexcitation maximum of 810 nm and an emission maximum of 830 nm.

In either the normal or augmented viewing modes, the light from thelight source or light sources is directed into an illumination channel216. Illumination channel 216 provides the light to another illuminationchannel in stereoscopic endoscope 201 that in turn directs the light tosurgical site 203.

The video output on stereoscopic display unit 241 may be toggled betweenthe normal and augmented viewing modes by using, e.g., a foot switch, adouble click of the master grips that control the surgical instruments,voice control, and other like switching methods. The toggle forswitching between the two viewing modes is represented in FIG. 2 asdisplay mode select unit 250.

In response to a user input from display mode select unit 250, a signalis provided to a VIEWING MODE check process 301 (FIG. 3) in a userinterface 260 that in turn provides a control signal to ILLUMINATETISSUE process 302 when the normal viewing mode is selected. Userinterface 260, in one aspect, is generated by computer code, which isstored in a memory 132, executing on a processor 131 (FIG. 1).

In one aspect, the normal viewing mode is a default mode. In thisaspect, display mode select unit 250 would not be used until the surgeonwanted to change the viewing mode from the normal viewing mode to theaugmented viewing mode or from the augmented viewing mode to the normalviewing mode.

In the normal viewing mode, ILLUMINATE TISSUE process 302 sends a normalviewing mode operation signal to power and level controller 215 incombination light source 210. Power and level controller 215 isillustrated in combination light source 210 for convenience and is notintended to limit the location of power and level controller 215 to thisspecific location.

In response to the normal viewing mode operation signal, power and levelcontroller 215 turns off fluorescence excitation source 212, if source212 is on, and enables visible color component source 211 so that whitelight is provided to surgical site 203. For example, when visible colorcomponent source 211 includes three visible color illumination componentsources, power is provided to each of the three sources. Thoseknowledgeable in the field recognize that instead of turning the poweron and off to the various sources in 210, controller 215 could maintainthe power always on and direct the output from the sources to and awayfrom channel 216 and achieve the same result.

Thus, in the normal viewing mode, ILLUMINATE TISSUE process 302 causessurgical site 203 to be illuminated with white light. In the graphs ofthe illumination in FIG. 4, the horizontal axis is time and the verticalaxis represents source output level. The source output level duringnormal viewing mode operation for each of the three visible colorillumination components is defined as 100 percent. Thus, in FIG. 4 fortimes before time t1, the output level from each of the three visiblecolor illumination components is shown as 100 percent and the outputlevel for the fluorescence excitation illumination component is zero.

The visible light from surgical site 203 (FIG. 2) is passed by thestereoscopic optical channels in endoscope 201 to image capture system220. Image capture system 220, in this aspect, includes a stereoscopicimage capture system that includes a left image capture unit 221A thatis a first single-chip image capture sensor with a color filter arrayand a right image capture unit 222A that is a second single-chip imagecapture sensor with a color filter array.

Thus, in CAPTURE RAW SCENE PER FRAME process 303 (FIG. 3) in the normalviewing mode, left image capture unit 221A captures a visible left scene421A (FIG. 4) in a frame and right image capture unit 222A captures avisible right scene 422A in a frame. Left image capture unit 221Acaptures red, green, and blue pixel data for visible left scene 421A,i.e., the acquired left scene is a color scene. Similarly, right imagecapture unit 222A captures red, green, and blue pixel data for visibleright image 422A.

In the normal viewing mode, acquired normal left visible scene 421A andacquired normal visible right scene 422A (FIG. 4) are provided todisplay image controller 230 (FIG. 2) that performs COLOR CORRECTIONprocess 304 (FIG. 3). When the augmented signal is false, i.e., in thenormal viewing mode, display image controller 230 couples the acquiredscenes in image capture system 220 to color correction module 231. Thus,in COLOR CORRECTION process 304, color correction module 231 processesboth acquired normal left visible scene 421A and acquired normal rightvisible scene 422A. The color corrected acquired normal left visiblescene and the color corrected acquired normal right visible are sent tostereoscopic display unit 241 and a stereoscopic full color scene isdisplayed in GENERATE A STEREOSCOPIC VIDEO DISPLAY OF TISSUE process305.

The processing in the normal viewing mode is equivalent to theprocessing in a conventional minimally invasive surgical system and sois known to those knowledgeable in the field. Also, processes 301 to 305are performed repetitively for each frame so that the surgeon sees areal-time video image of surgical site 203.

During the normal viewing mode, the surgeon is provided with a normalthree-dimensional color view 600A of surgical site 203A (FIG. 6A).However, the surgeon may wish to see a region or regions of interest insurgical site 203A highlighted in the three-dimensional view of surgicalsite 203A. For example, the surgeon may which to see diseased portionsof surgical site 203A and/or a specific tissue, e.g., a tendon or organ.Thus, at time t1 (FIG. 4), the surgeon uses display mode select unit 250to change the viewing mode to the augmented viewing mode.

In response to the user input from display mode select unit 250, anaugmented display selection signal is provided to a VIEWING MODE checkprocess 301 in user interface 260. In response to the augmented displayselection signal, check process 301 provides an augmented imagingcontrol signal to ILLUMINATE TISSUE process 302 and to COLOR CORRECTIONprocess 304.

In response to the augmented display control signal, ILLUMINATE TISSUEprocess 302 sends an augmented display signal having a state of true topower and level controller 215 in combination light source 210. Inresponse to the augmented display signal having state true, power andlevel controller 215 turns on fluorescence excitation source 212 and inthis example turns off the first and third visual color illuminationcomponents in visible color component source 211 so that only the secondvisual color illumination component and the fluorescence excitationillumination component are supplied to illumination channel 216. Thus,surgical site 203 is illuminated with the second visual colorillumination component, but not with the first and third visual colorillumination components. Surgical site 203 is also illuminated with thefluorescence excitation illumination component.

Also, in one embodiment, power and level controller 215 reduces theoutput level of the second visual color illumination component, e.g.,reduces the output level to one part in ten. Thus, as shown in FIG. 4,after time t1, the output level of the second visual color illuminationcomponent is reduced relative to the output level prior to time t1, andthe first and third visual color component illumination output levelsare zero. Also, the fluorescence excitation illumination component isturned on. As explained more completely below, the output level isreduced to avoid saturation of pixels and to maintain a proper contrastbetween the acquired visible scene component or components and theacquired fluorescence scene component.

The light from surgical site 203 (FIG. 2) is passed by the stereoscopicleft and right optical channels in endoscope 201 to image capture system220. In one aspect, filters 221B, 222B are used to filter any reflectedor direct light from fluorescence excitation source 212 before thescenes are captured.

Prior to considering CAPTURE RAW SCENE PER FRAME process 303 (FIG. 3), asingle-chip image capture sensor with a color filter array is brieflyconsidered. Many different types of color filter arrays are known. Acommonly used color filter array is a Bayer color filter array. For athree visible color component color space (C1, C2, C3), a Bayer colorfilter array has about twice as many filters for one visible colorcomponent as for each of the other two color components. For example, ina six by six pixel block, one implementation of the Bayer color filterarray has eighteen filters for color component C2, nine filters forcolor component C1, and nine filters for color component C3. The layoutof the color filters in the Bayer pattern is well known and so is notdescribed further herein.

The use of a three color component Bayer filter is illustrative only andis not intended to be limiting. Other types of color filter arrays canbe used such as, for example, a red-green-blue-emerald color filterarray.

In CAPTURE RAW SCENE PER FRAME process 303 (FIG. 3), in the augmentedviewing mode, left image capture unit 221A captures a frame thatincludes a plurality of sets of pixel data. Each set of pixel data canalso include captured light from colors components in the visiblespectrum adjacent to the color of the color filter. This captured lightleaked through the color filter for that set of pixels. In this example,the visible color component illumination is for the second colorcomponent.

Thus, in left image capture unit 221A, a first set of pixel data forfirst color component C1 is a first combined scene 421B (FIG. 4) that isa first weighted combination of a left visible second color componentscene, i.e., the reflected second color component illumination thatleaked through the first color component filter, and a left fluorescencescene component. A second set of pixel data for second color componentC2 is a second combined scene 421C (FIG. 4) that is a second weightedcombination of the left visible second color component scene and theleft fluorescence scene component. A third set of pixel data for a thirdcolor component C3 is a third combined scene 421D (FIG. 4) that is athird weighted combination of a left visible second color componentscene, i.e., the reflected second color component illumination thatleaked through the third color component filter, and a left fluorescencescene component. Note that the weights in each of the combined scenesare determined by the characteristics of the filter in the color filterarray for that color component.

Right image capture unit 222A also captures a frame that includes aplurality of sets of pixel data. Again, each set of pixel data can alsoinclude captured light from colors components in the visible spectrumadjacent to the color of the color filter. This captured light leakedthrough the color filter for that set of pixels. In this example, thevisible color component illumination is second color componentillumination.

Thus, in right image capture unit 222A, a first set of pixel data forfirst color component C1 is a first combined scene 422B (FIG. 4) that isa first weighted combination of a right visible second color componentscene, i.e., the reflected second color component illumination thatleaked through the first color component filter, and a rightfluorescence scene component. A second set of pixel data for secondcolor component C2 is a second combined scene 422C (FIG. 4) that is asecond weighted combination of the right visible second color componentscene and the right fluorescence scene component. A third set of pixeldata for a third color component C3 is a third combined scene 422D (FIG.4) that is a third weighted combination of the right visible secondcolor component scene, i.e., the reflected second color componentillumination that leaked through the third color component filter, and aright fluorescence scene component. As noted above, the weights in eachof the combined scenes are determined by the characteristics of thefilter in the color filter array for that color component.

No additional cameras, optic channels in the endoscope, or additionalendoscopes are needed to acquire both the visible color component sceneand the fluorescence scene component. Herein, when it is stated that avisible color component scene in a scene is associated with a visiblecolor component illumination source, it means that the visible colorcomponent illumination source provides the light that results in thatvisible color component scene in the image.

Recall, as described above, display scene controller 230 has receivedthe augmented display signal having state true. Thus, display scenecontroller 230 changes from color image correction module 231 toaugmented color correction module 232. In the augmented viewing mode,the acquired raw scene data is provided to augmented color correctionmodule 232 that performs COLOR CORRECTION process 304. Hence, in oneaspect, COLOR CORRECTION process 304 is used in both color imagecorrection module 231 and augmented color correction module 232. Theprocessing performed by COLOR CORRECTION process 304 depends on themodule that is using COLOR CORRECTION process 304.

COLOR CORRECTION process 304, as implemented by augmented colorcorrection module 232, performs the same process on both the left andright acquired pixel data, and so the left and right designation is notconsidered in the description. FIG. 5 is a block diagram of the inputinformation to augmented color correction module 232, and the outputinformation from augmented color correction module 232.

A scene processing module 504 in augmented color correction module 232performs an augmented color correction process as process 304. Theaugmented color correction process receives on a first input colorchannel CC1 the acquired pixel data including first combined scene λ1that includes fluorescence scene component λ, on a second input colorchannel CC2 acquired pixel data including the second combined scene ofsecond visible color component scene AC2 and fluorescence scenecomponent λ, and on a third input color channel CC3 acquired pixel dataincluding third combined scene λ3 that includes fluorescence scenecomponent λ. The augmented color correction process scales the colorchannel inputs to compensate for any differences in the responses of thecolor filters in the color filter array to the fluorescence.

Next, the augmented color correction process, as explained morecompletely below, demosaics the acquired pixel data to generate a fullresolution array of image pixel data for at least two of the three colorcomponents—a first full resolution array of image pixel data for firstcolor component C1 includes first combined scene λ1 that includesfluorescence scene component λ. A second full resolution array of imagepixel data for the second color component C2 includes the secondcombined scene that includes acquired second visible color componentscene AC2 combined with acquired fluorescence scene component λ. A thirdfull resolution array of image pixel data for third color component C3includes third combined scene λ3 that includes fluorescence scenecomponent λ.

Scene processing module 504 generates a display visible scene componentDVSC by using the first full resolution array of image pixel dataincluding fluorescence scene component λ to remove the contribution offluorescence scene component λ from the second full resolution array ofimage pixel data. More specifically, scene processing module 504generates display visible scene component DVSC using a weighted linearcombination of the image pixel data in the first and second fullresolution arrays of image pixel data. Scene processing module 504 alsogenerates a display fluorescence scene component DFSC using a weightedlinear combination of the image pixel data in the first and second fullresolution arrays.

Finally, scene processing module 504 generates a weighted combination ofdisplay visible scene component DVSC and display fluorescence scenecomponent DFSC for each input color channel of stereoscopic display unit241. In the example of FIG. 5, stereoscopic display unit 241 has threeinput color channels. Thus, scene processing module 504 outputs threevisible color channel inputs which are weighted combinations of displayvisible scene component DVSC and display fluorescence scene componentDFSC. In FIG. 5, the weights are W₁₁, W₁₂, W₂₁, W₂₂, W₃₁, and W₃₂.

Consider the example, where W₁₁=W₂₁=W₃₁=W₂₂=1 and W₁₂=W₃₂=augmentedcolor correction module 232, in this example, adds display fluorescencescene component DFSC to display visible scene component DVSC. The resultof the addition is supplied to the second output color channel. In thisexample, display visible scene component DVSC is sent to each of thefirst and third output color channels. Those knowledgeable in the fieldunderstand that the operations described with respect to the augmentedcolor correction process are done with respect to a subunit of a frame,e.g., on a pixel by pixel basis and that the “addition” is symbolic andmay require undoing and redoing gamma correction for example to achievea clear image.

The outputs from augmented color correction module 232 are displayed onstereoscopic display 241 (FIG. 2) in GENERATE STEREOSCOPIC VIDEO DISPLAYOF TISSUE process 305 (FIG. 3). In the augmented viewing mode, processes301 to 305 are performed repetitively on each acquired frame so that thesurgeon sees a real-time video augmented image of surgical site 203.During the augmented viewing mode, the surgeon is provided with athree-dimensional reduced color scene 600B of surgical site 203B withregion of interest 603 (FIG. 6B) highlighted in a particular color.

FIG. 7 is a more detailed diagram of one aspect of scene processingmodule 504. Scene processing module 504 includes a demosaic module 701,a scene component generator 702, and a display color component generator703.

FIGS. 8A and 8B are illustrations of input information to demosaicmodule 701 and output information from demosaic module 701. In FIG. 8A,acquired pixel data 802 is representative of a frame of pixel datacaptured by a single-chip image capture sensor having a Bayer colorfilter array. Again, in this example, three visible color components C1,C2, and C3 are used. Interpolation techniques used in demosaicing areknown to those knowledgeable in the field and so are not considered infurther detail.

A pixel in acquired pixel data 802 is represented by a square in FIGS.8A and 8B. The color component in a square gives the color of the filterin the color filter array for that pixel, i.e., one of color C1, C2, andC3. The information enclosed within parentheses in a square is the lightthat is captured by the pixel in this example.

Since the fluorescence is in the near infrared spectrum, thefluorescence is passed through the filters in the color filter array foreach of three visible color components C1, C2, and C3. Thus, each of thepixels captures the brightness of fluorescence scene component λ at thelocation of the pixel.

In this example, the visible color illumination component is colorcomponent C2. There is no illumination for visible color components C1and C3. Light reflected from the tissue is color component C2 light andis passed through the filters in the color filter array that pass colorcomponent C2 light. The second visible color component C2 light iscaptured by the image capture sensor is acquired second visible colorcomponent scene AC2 in FIGS. 8A and 8B. Note that pixels for colorcomponents C1 and C3 can also capture some of the reflected colorcomponent C2 light if the reflected light leaks through the filters forcolor components C1 and C3 in the color filter array. This is why thescenes for color components C1 and C3 are referred to as combined scenesλ1 and λ3, respectively.

As illustrated in FIGS. 7, 8A, and 8B, in this aspect, demosaic module701 receives a third set of pixel data 813 of color component C3 in thethird color channel, and a first set of pixel data 811 of colorcomponent C1 in a first color channel. Third set of pixel data 813includes third combined image λ3 that in turn includes fluorescencescene component λ. First set of pixel data 811 includes first combinedimage λ1 that in turn includes fluorescence scene component λ. Demosaicmodule 701 also receives, in the second color channel, the secondcombined image that includes a combination of acquired second visiblecolor component scene AC2 and fluorescence scene component λ in a secondset of pixel data 812 of color component C2.

Using the received pixel data in each color channel, in the aspect ofFIG. 8A, demosaic module 701 generates a full resolution set of imagepixel data for each of the three color components, which are representedby arrays 821 to 823. However, in one aspect, when only a single visiblecolor component illumination source is used with a single fluorescenceexcitation illumination component, as in this example, only one of fullresolution image pixel arrays for color components C1 and C3 is neededand so only the acquired pixel data for one of color components C1 andC3 is demosaiced. However, if a second visible color componentillumination source is used or alternatively, a second fluorescence witha spectrum in the range of one of color component C1 and C3 is captured,the full demosaic process illustrated in FIG. 8A is used.

In another aspect, when only a single visible color componentillumination is used with a single fluorescence excitation illuminationcomponent, demosaic module 701 combines the acquired pixel data of thetwo color components that includes fluorescence scene component λ andleakage data, and processes the combined pixel data as single colorchannel to generate a full resolution set of image pixel data comprisinga fourth combined scene. In the example of FIG. 8A, the acquired datafor pixels of color components C1 and C3 includes fluorescence scenecomponent λ and leakage data. Thus, demosaic module 701 combines thepixel data in channels 811 and 813 in FIG. 8A and creates combined pixeldata for a color channel 831. Demosaic module 701 processes the acquireddata for color components C1 and C3 as a single color channel 831 andgenerate a full resolution set of image pixel data comprising a fourthcombined scene λ31 (FIG. 8B). The pixel data in color channel 831 isillustrated in FIG. 8B as having color component C31.

Thus, in this aspect, two color channels 812, 831 are demosaiced toobtain a full resolution pixel array for each of the color channels.Since a larger number of acquired data locations are used in thedemosaicing of the combined color channel, the accuracy of the fullresolution data is improved relative to when only pixel data for asingle color channel in the combined color channel is used in thedemosaicing.

The transmission of the fluorescence through the different colorcomponent filters in the color filter array may not be the same. Thus,in one aspect, correction factors are used to scale the acquired pixeldata for each color component so that the values that are demosaiced areequal for equal fluorescence. The correction factors are experimentallydetermined by capturing light from a known light source using the imagecapture sensor with the color filter array. The captured calibrationpixel data is analyzed and scale factors determined so that the responseis equal for each color component for equal fluorescence.

In this example, scene component generator 702 receives a first fullresolution array ACC1 of image pixel data of combined scene λ1 thatincludes acquired fluorescence scene component λ, e.g., one of array 821and array 841, in a first input color channel CC1. Scene componentgenerator 702 also receives a second full resolution array ACC2 of imagepixel data of the second combined scene that is a combination ofacquired second visible color component scene AC2 and fluorescence scenecomponent λ, i.e., array 822, in a second input color channel CC2.

Scene component generator 702 generates a display visual scene componentDVSC and a display fluorescence scene component DFSC from the two fullresolution arrays of image pixel data In one aspect, the extraction isrepresented as:DFSC(x,y)=ACC1(x,y)−s2*ACC2(x,y)  (1)DVSC(x,y)=ACC2(x,y)−s1*ACC1(x,y)  (2)

In general terms, display visual scene component DVSC is linear weightedcombination of acquired second visible color component scene AC2 andfluorescence scene component λ. Display fluorescence scene componentDFSC is a different linear weighted combination of acquired secondvisible color component scene AC2 and fluorescence scene component λ.

Definitions (1) and (2) indicate that the display visual scene componentDVSC at location (x, y) is a weighted linear combination of the imagepixel value in the first color channel and the image pixel value in thesecond color channel at location (x, y), where the image pixel value inthe second color channel is a combination of the acquired fluorescenceand visible scene components (AC2+λ) and the image pixel value in thefirst color channel is the first combined scene that includes acquiredfluorescence scene component λ at location (x, y) combined with anyleakage of second visible color component scene AC2 at location (x, y).Similarly, the display fluorescence scene component DFSC at location (x,y) is a weighted linear combination of the image pixel value in thefirst color channel and the image pixel value in the second colorchannel at location (x, y). Scale factors s1 and s2 used in theextraction can be determined as those described more completely below.

In this example, display color component generator 703 receives a fullresolution display visual scene component DVSC and a full resolutiondisplay fluorescence scene component DFSC. Display color componentgenerator 703 generates a final display color component for each of thecolor channel inputs to the display unit. Thus, a plurality of finaldisplay color components is generated. As explained above, each finaldisplay color component is a weighted linear combination of fullresolution display visual scene component DVSC and a full resolutiondisplay fluorescence scene component DFSC.

In the example of FIG. 9, the display unit has n color component inputs,where n is a positive non-zero integer number. In the above example, nwas taken as three. The weights w_(ij) used in display color componentgenerator 703 are selected to provide a display scene that includes areduced color visible scene component combined with a fluorescence scenecomponent with color characteristics that allow easy differentiationbetween the two scenes. Multiple factors can be taken into account inselecting the weights such as brightness, contrast, color separation,eye fatigue, etc.

Below, an example is presented of obtaining scale factors for use inscene component generator 702. In this example, the response of theimage capture unit—the single-chip image capture sensor and color filterarray-is determined for visible illumination VIS and for fluorescenceexcitation illumination NIR, separately. The response is defined as:

$\begin{matrix}{{C_{VIS}\left( {x,y} \right)} = {\begin{bmatrix}{{CC}_{1}\left( {x,y} \right)} \\{{CC}_{2}\left( {x,y} \right)} \\\ldots \\{{CC}_{n}\left( {x,y} \right)}\end{bmatrix} = {\begin{bmatrix}S_{11} & S_{12} & S_{13} & \ldots & S_{1\; m} \\S_{21} & S_{22} & S_{23} & \ldots & S_{2\; m} \\\ldots & \ldots & \ldots & \ldots & \ldots \\S_{n\; 1} & S_{n\; 2} & S_{n\; 3} & \ldots & S_{nm}\end{bmatrix}*\begin{bmatrix}{E_{{VIS}\; 1}\left( {x,y} \right)} \\{E_{{VIS}\; 2}\left( {x,y} \right)} \\\ldots \\{E_{VISm}\left( {x,y} \right)}\end{bmatrix}}}} & (3) \\{{C_{NIR}\left( {x,y} \right)} = {\begin{bmatrix}{{CC}_{1}\left( {x,y} \right)} \\{{CC}_{2}\left( {x,y} \right)} \\\ldots \\{{CC}_{n}\left( {x,y} \right)}\end{bmatrix} = {\begin{bmatrix}S_{11} & S_{12} & S_{13} & \ldots & S_{1\; m} \\S_{21} & S_{22} & S_{23} & \ldots & S_{2\; m} \\\ldots & \ldots & \ldots & \ldots & \ldots \\S_{n\; 1} & S_{n\; 2} & S_{n\; 3} & \ldots & S_{nm}\end{bmatrix}*\begin{bmatrix}{E_{{NIR}\; 1}\left( {x,y} \right)} \\{E_{{NIR}\; 2}\left( {x,y} \right)} \\\ldots \\{E_{NIRm}\left( {x,y} \right)}\end{bmatrix}}}} & (4)\end{matrix}$where

-   -   C(x, y) represents the n color component response of the        single-chip image sensor,    -   S represents the image sensor responsivity functions,    -   E represents the spectral reflected light from the scene using        visible illumination, or fluorescence from the scene excited by        fluorescence excitation illumination,    -   m is the number of discrete wavelengths used to represent the        wavelength distribution of the reflected light in a plot of        reflected light intensity versus wavelength,    -   VIS represents visible illumination, and    -   NIR represents fluorescence excitation illumination.        Generating the image sensor responsivity functions is done using        techniques know to those knowledgeable in the field and so is        not considered in further detail.

A separation color matrix SCM is used in scene component generator 702to extract display visible scene component DVSC and display fluorescencescene component DVSC from the received image pixel data. Separationcolor matrix SCM is defined as:SCM(x,y)=pinv([C _(VIS)(x,y),C _(NIR)(x,y)])  (5)where pinv( ) is the pseudo inverse, and

-   -   [C_(VIS)(x, y), C_(NIR)(x, y)] is the concatenation of the two        column vectors into an [n×2] matrix.

A matrix pseudo inverse is known to those knowledgeable in the field. Apseudo inverse suitable to use here is referred to as the Moore-Penrosepseudo inverse. A common use of the Moore-Penrose pseudo inverse is tocompute a ‘best fit’ least squares solution to the system of linearequations, which lacks a unique solution. Another use of theMoore-Penrose pseudo inverse is to find the minimum (Euclidean) normsolution to the system of linear equations. In one aspect, the best fitleast squares solution is used. However, other inverse methods existthat can lower the noise amplification of the separation color matrix.

Thus, in this aspect, the processing implemented in scene componentgenerator 702 is:

$\begin{matrix}{\begin{bmatrix}{{DVSC}\left( {x,y} \right)} \\{{DFSC}\left( {x,y} \right)}\end{bmatrix} = {\begin{bmatrix}{scm}_{11} & {scm}_{12} & \ldots & {scm}_{1\; k} \\{scm}_{21} & {scm}_{22} & \ldots & {scm}_{2\; k}\end{bmatrix}*\begin{bmatrix}{{ACC}\; 1\left( {x,y} \right)} \\{{ACC}\; 2\left( {x,y} \right)} \\\ldots \\{{ACCk}\left( {x,y} \right)}\end{bmatrix}}} & (6)\end{matrix}$where

-   -   ACCi(x, y) is a value of an image pixel at location (x, y) in        the full resolution array for the ith color channel from        demosaic module 701, where i ranges from 1 to k and k is the        number of input color channels to scene component generator 702        that receive a full resolution array of image pixel data from        demosaic module 701,    -   scm_(ij) is a weight in the separation color matrix SCM,    -   DVSC(x, y) is a value of a pixel at location (x, y) in the        display visual scene component DVSC, and    -   DFSC(x, y) is a value of a pixel at location (x, y) in the        display fluorescence scene component DFSC.

In above example, the number of input color channel was taken as threeand so

$\begin{matrix}{\begin{bmatrix}{{DVSC}\left( {x,y} \right)} \\{{DFSC}\left( {x,y} \right)}\end{bmatrix} = {\begin{bmatrix}{scm}_{11} & {scm}_{12} & {scm}_{13} \\{scm}_{21} & {scm}_{22} & {scm}_{23}\end{bmatrix}*\begin{bmatrix}{{ACC}\; 1\left( {x,y} \right)} \\{{ACC}\; 2\left( {x,y} \right)} \\{{ACC}\; 3\left( {x,y} \right)}\end{bmatrix}}} & (7) \\{\begin{bmatrix}{{DVSC}\left( {x,y} \right)} \\{{DFSC}\left( {x,y} \right)}\end{bmatrix} = {\begin{bmatrix}{scm}_{11} & {scm}_{12} & {scm}_{13} \\{scm}_{21} & {scm}_{22} & {scm}_{23}\end{bmatrix}*\begin{bmatrix}{\left\{ {\lambda + {{fw}\; 1*{AC}\; 2}} \right\}\left( {x,y} \right)} \\{\left\{ {{{AC}\; 2} + \lambda} \right\}\left( {x,y} \right)} \\{\left\{ {\lambda + {{fw}\; 3*{AC}\; 2}} \right\}\left( {x,y} \right)}\end{bmatrix}}} & (8)\end{matrix}$where (fw1*AC2) represents second color component light that leaksthrough the filter for the first color component in the color filterarray, and (fw3*AC2) represents second color component light that leaksthrough the filter for the third color component in the color filterarray.

Thus, at location (x, y),

$\begin{matrix}{\begin{matrix}{{DVSC} = {{{scm}_{11}*\left\{ {\lambda + {{fw}\; 1*{AC}\; 2}} \right\}} + {{scm}_{12}*}}} \\{\left\{ {{{AC}\; 2} + \lambda} \right\} + {{scm}_{13}*\left\{ {\lambda + {{fw}\; 3*{AC}\; 2}} \right\}}} \\{= {\left( {{{scm}_{11}*{fw}\; 1} + {scm}_{12} + {{scm}_{13}*{fw}\; 3}} \right)*}} \\{{{AC}\; 2} - {\left( {{- {scm}_{11}} - {scm}_{12} - {scm}_{13}} \right)*\lambda}}\end{matrix}} & (9) \\\begin{matrix}{{DFSC} = {{{scm}_{21}*\left\{ {\lambda + {{fw}\; 1*{AC}\; 2}} \right\}} + {{scm}_{22}*\left\{ {{{AC}\; 2} + \lambda} \right\}} +}} \\{{scm}_{23}*\left\{ {\lambda + {{fw}\; 3*{AC}\; 2}} \right\}} \\{= {{\left( {{scm}_{21} + {scm}_{22} + {scm}_{23}} \right)*\lambda} -}} \\{\left( {{{- {scm}_{21}}*{fw}\; 1} - {scm}_{22} - {{scm}_{23}*{fw}\; 1}} \right)*{AC}\; 2}\end{matrix} & (10)\end{matrix}$

In above expression (7), the weights in separation color matrix SCM havebeen taken as positive values, but in practice some are negative. Also,the display components can be normalized so that these expressions arein the same form as those given above in expressions (1) and (2). Thus,the contribution of the acquired fluorescence scene component is removedfrom the displayed visual scene component DVSC and the contribution ofthe acquired second visible color component scene AC2 is removed fromthe displayed fluorescence scene component DFSC by scene componentgenerator 702.

In the above example, a single visible color illumination component anda single fluorescence excitation illumination component were used. Asstated previously, the processes described above work with a single-chipimage capture sensor with a color filter array and other combinations ofillumination. For example, for n visible color components that make upwhite light, some of the possible combination are:

-   -   (n−1) or less visible color illumination components and a        fluorescence illumination excitation component, where the        fluorescence is either outside the visible spectrum, or in the        wavelength range of the nth color component, where the nth color        component is the one for which no visible illumination was        provided; and    -   (n−2) or less visible color illumination components and two        different fluorescence illumination excitation components that        excite fluorescence either in the wavelength range of the nth        and (n−1) visible color components, where the nth and (n−1)        color components are the ones which contain no visible        illumination, or that excite in the wavelength range of the nth        visible color component and outside the visible spectrum, where        the nth color component is the one that contains no visible        illumination.

The processing of the data for these combinations is similar to thatdescribed above, except extra components must be considered thatcorrespond to the additional pixel data that is captured. Also, theleakage characteristics of the color filter array are considered todetermine the extra components that must be captured.

For example if there are red, green, and blue color components and thefluorescence is in the blue spectrum, the blue pixel data represents thefluorescence scene component. The green pixel data represents acombination of a green scene component and the fluorescence scenecomponent. The red pixel data represents a red scene component. Theacquired data is processed as described above, except for this example,the red pixel data does not include fluorescence and so the leakageweight for the fluorescence in the combined scene for the red colorcomponent is zero.

All examples and illustrative references are non-limiting and should notbe used to limit the claims to specific implementations and embodimentsdescribed herein and their equivalents. The headings are solely forformatting and should not be used to limit the subject matter in anyway, because text under one heading may cross reference or apply to textunder one or more headings. Finally, in view of this disclosure,particular features described in relation to one aspect or embodimentmay be applied to other disclosed aspects or embodiments of theinvention, even though not specifically shown in the drawings ordescribed in the text. For example, a surgical system is used as anexample. However, the mixed mode imaging implemented using a single-chipimage capture sensor with a color filter array as described herein canbe implemented in any system that includes the components, assemblies,etc. used in the mixed mode imaging.

The various modules described herein can be implemented by softwareexecuting on a processor, hardware, firmware, or any combination of thethree. When the modules are implemented as software executing on aprocessor, the software is stored in a memory as computer readableinstructions and the computer readable instructions are executed on theprocessor. All or part of the memory can be in a different physicallocation than a processor so long as the processor can be coupled to thememory. Memory refers to a volatile memory, a non-volatile memory, orany combination of the two.

Also, the functions of the various modules, as described herein, may beperformed by one unit, or divided up among different components, each ofwhich may be implemented in turn by any combination of hardware,software that is executed on a processor, and firmware. When divided upamong different components, the components may be centralized in onelocation or distributed across system 200 for distributed processingpurposes. The execution of the various modules results in methods thatperform the processes described above for the various modules andcontroller 130.

Thus, a processor is coupled to a memory containing instructionsexecuted by the processor. This could be accomplished within a computersystem, or alternatively via a connection to another computer via modemsand analog lines, or digital interfaces and a digital carrier line.

Herein, a computer program product comprises a computer readable mediumconfigured to store computer readable code needed for any part of or allof the processes described herein, or in which computer readable codefor any part of or all of those processes is stored. Some examples ofcomputer program products are CD-ROM discs, DVD discs, flash memory, ROMcards, floppy discs, magnetic tapes, computer hard drives, servers on anetwork and signals transmitted over a network representing computerreadable program code. A non-transitory tangible computer programproduct comprises a non-transitory tangible computer readable mediumconfigured to store computer readable instructions for any part of orall of the processes or in which computer readable instructions for anypart of or all of the processes is stored. Non-transitory tangiblecomputer program products are CD-ROM discs, DVD discs, flash memory, ROMcards, floppy discs, magnetic tapes, computer hard drives and otherphysical storage mediums.

In view of this disclosure, instructions used in any part of or all ofthe processes described herein can be implemented in a wide variety ofcomputer system configurations using an operating system and computerprogramming language of interest to the user.

Herein, first and second are used as adjectives to distinguish betweenelements and are not intended to indicate a number of elements. Also,top, bottom, and side are used as adjectives to aid in distinguishingbetween elements as viewed in the drawings, and to help visualizerelative relationships between the elements. For example, top and bottomsurfaces are first and second surfaces that are opposite and removedfrom each other. A side surface is a third surface that extends betweenthe first and second surfaces. Top, bottom, and side are not being usedto define absolute physical positions.

The above description and the accompanying drawings that illustrateaspects and embodiments of the present inventions should not be taken aslimiting—the claims define the protected inventions. Various mechanical,compositional, structural, electrical, and operational changes may bemade without departing from the spirit and scope of this description andthe claims. In some instances, well-known circuits, structures, andtechniques have not been shown or described in detail to avoid obscuringthe invention.

Further, this description's terminology is not intended to limit theinvention. For example, spatially relative terms—such as “beneath”,“below”, “lower”, “above”, “upper”, “proximal”, “distal”, and thelike—may be used to describe one element's or feature's relationship toanother element or feature as illustrated in the figures. Thesespatially relative terms are intended to encompass different positions(i.e., locations) and orientations (i.e., rotational placements) of thedevice in use or operation in addition to the position and orientationshown in the figures. For example, if the device in the figures isturned over, elements described as “below” or “beneath” other elementsor features would then be “above” or “over” the other elements orfeatures. Thus, the exemplary term “below” can encompass both positionsand orientations of above and below. The device may be otherwiseoriented (rotated 90 degrees or at other orientations) and the spatiallyrelative descriptors used herein interpreted accordingly. Likewise,descriptions of movement along and around various axes include variousspecial device positions and orientations.

The singular forms “a”, “an”, and “the” are intended to include theplural forms as well, unless the context indicates otherwise. The terms“comprises”, “comprising”, “includes”, and the like specify the presenceof stated features, steps, operations, elements, and/or components butdo not preclude the presence or addition of one or more other features,steps, operations, elements, components, and/or groups. Componentsdescribed as coupled may be electrically or mechanically directlycoupled, or they may be indirectly coupled via one or more intermediatecomponents.

What is claimed is:
 1. A system comprising: an image capture unitincluding a single-chip image capture sensor with a color filter array,wherein the single-chip image capture sensor is configured to capture aframe of pixel data comprising a first set of pixel data and a secondset of pixel data, wherein the first set of pixel data comprises a firstcombined scene including a fluorescence scene component, and wherein thesecond set of pixel data comprises a second combined scene including acombination of a visible color component scene and the fluorescencescene component; a scene processing module configured to: receive thefirst set of pixel data comprising the first combined scene and thesecond set of pixel data comprising the second combined scene, extract adisplay fluorescence scene component from the first combined scene,extract a display visible scene component from the second combinedscene, and generate a plurality of weighted combinations of the displayfluorescence scene component and the display visible scene component;and a display unit connected to the scene processing module to receivethe plurality of weighted combinations, and configured to generate fromthe plurality of weighted combinations a displayed scene including ahighlighted scene component corresponding to the fluorescence scenecomponent and a reduced color scene component.
 2. The system of claim 1,further comprising: an illuminator configured to provide at least twoillumination components: wherein one of the illumination components is afluorescence excitation illumination component, wherein otherillumination components include less than all visible color componentsof white light, and wherein the at least two illumination components areprovided at the same time.
 3. The system of claim 2, the illuminatorfurther comprising: a fluorescence excitation illumination source; and avisible color component illumination source.
 4. The system of claim 1,wherein the visible color component scene is captured as green colorpixels in the single-chip image capture sensor.
 5. The system of claim3, further comprising: a power level and power supply controllerconnected to the illuminator; and a mode changer, coupled to the powerlevel and power supply controller, having a first state and a secondstate, wherein, when the mode changer has the first state, the powerlevel and power supply controller (a) provides power to the visiblecolor component illumination source, and not to the fluorescenceexcitation illumination source, and (b) the visible color componentillumination source has a first level of illumination; and wherein, whenthe mode changer has the second state, the power level and power supplycontroller (a) provides power to the visible color componentillumination source and to the fluorescence excitation illuminationsource, and (b) reduces the level of illumination of first visible colorcomponent illumination source relative to a level of illumination of thevisible color component illumination source in the first state.
 6. Thesystem of claim 1, the scene processing module further comprising: ademosaic module coupled to the image capture unit to receive, in a firstcolor channel, the first set of pixel data comprising the first combinedscene, wherein the demosaic module is configured to demosaic the firstset of pixel data to obtain a first set of image pixel data comprisingthe first combined scene.
 7. The system of claim 6, wherein the demosaicmodule is coupled to the image capture unit to receive, in a secondcolor channel, the second set of pixel data comprising the secondcombined scene, and wherein the demosaic module is configured todemosaic the second set of pixel data to obtain a second set of imagepixel data comprising the second combined scene.
 8. The system of claim1, the scene processing module further comprising: a demosaic modulecoupled to the image capture unit to receive, in a first color channel,the first set of pixel data comprising the first combined scene, and toreceive, in a third color channel, a third set of pixel data comprisinga third combined scene including the fluorescence scene component,wherein the demosaic module is configured to demosaic the first andthird pluralities of pixel data as a single color channel to obtain afirst set of image pixel data comprising a fourth combined scenecomponent including the fluorescence scene component.
 9. The system ofclaim 8, wherein the demosaic module is coupled to the image captureunit to receive, in a second color channel, the second set of pixel datacomprising the second combined scene, and wherein the demosaic module isconfigured to demosaic the pixel data representing the second combinedscene to obtain a second set of image pixel data comprising the secondcombined scene.
 10. The system of claim 1, wherein the scene processingmodule further comprises: a scene component generator configured to:receive a first set of image pixel data comprising the first combinedscene and a second set of image pixel data comprising the secondcombined scene, perform the extraction of the display fluorescence scenecomponent, wherein the display fluorescence scene component comprises afirst linear weighted combination of the first set of image pixel dataand the second set of image pixel data, and perform the extraction ofthe display visible scene component, wherein the display visible scenecomponent comprises a second linear weighted combination of the firstset of image pixel data and the second set of image pixel data.
 11. Amethod comprising: receiving a frame captured by a single-chip imagecapture sensor having a color filter array, wherein the frame comprisesa first set of pixel data and a second set of pixel data, wherein thefirst set of pixel data comprises a first combined scene including afluorescence scene component, and wherein the second set of pixel datacomprises a second combined scene including a combination of a visiblecolor component scene and the fluorescence scene component; extracting adisplay fluorescence scene component from the first combined scene;extracting a display visible scene component from the second combinedscene; generating a plurality of weighted combinations of the displayfluorescence scene component and the display visible scene component;and generating from the plurality of weighted combinations a displayedscene including a highlighted scene component corresponding to thefluorescence scene component and a reduced color scene component. 12.The method of claim 11 further comprising: illuminating a surgical sitewith at least two illumination components: wherein one of theillumination components is a fluorescence excitation illuminationcomponent, wherein other illumination components include less than allvisible color components of white light, and wherein the at least twoillumination components are provided at the same time.
 13. The method ofclaim 11 further comprising: demosaicing the first set of pixel datacomprising the first combined scene to obtain a first set of image pixeldata comprising the first combined scene, and demosaicing the second setof pixel data comprising the second combined scene to obtain a secondset of image pixel data comprising the second combined scene.
 14. Themethod of claim 11 further comprising: demosaicing the first set ofpixel data comprising the first combined scene and a third set of pixeldata comprising a third combined scene as a single color channel toobtain a first set of image pixel data including a fourth combined scenecomponent; and demosaicing the second set of pixel data comprising thesecond combined scene to obtain a second set of image pixel datacomprising the second combined scene.
 15. The method of claim 11 furthercomprising: receiving a first set of image pixel data comprising thefirst combined scene and a second set of image pixel data comprising thesecond combined scene, performing the extraction of the displayfluorescence scene component, wherein the display fluorescence scenecomponent comprises a first linear weighted combination of the first setof image pixel data and the second set of image pixel data, andperforming the extraction of the display visible scene component,wherein the display visible scene component comprises a second linearweighted combination of the first set of image pixel data and the secondset of image pixel data.